<!-------- @HEADER
 !
 ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 !
 !  Zoltan Toolkit for Load-balancing, Partitioning, Ordering and Coloring
 !                  Copyright 2012 Sandia Corporation
 !
 ! Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
 ! the U.S. Government retains certain rights in this software.
 !
 ! Redistribution and use in source and binary forms, with or without
 ! modification, are permitted provided that the following conditions are
 ! met:
 !
 ! 1. Redistributions of source code must retain the above copyright
 ! notice, this list of conditions and the following disclaimer.
 !
 ! 2. Redistributions in binary form must reproduce the above copyright
 ! notice, this list of conditions and the following disclaimer in the
 ! documentation and/or other materials provided with the distribution.
 !
 ! 3. Neither the name of the Corporation nor the names of the
 ! contributors may be used to endorse or promote products derived from
 ! this software without specific prior written permission.
 !
 ! THIS SOFTWARE IS PROVIDED BY SANDIA CORPORATION "AS IS" AND ANY
 ! EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 ! IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 ! PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SANDIA CORPORATION OR THE
 ! CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 ! EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 ! PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 ! PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 ! LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 ! NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 ! SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 !
 ! Questions? Contact Karen Devine	kddevin@sandia.gov
 !                    Erik Boman	egboman@sandia.gov
 !
 ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 !
 ! @HEADER
-------> 


<HTML>
<HEAD>
   <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
   <META NAME="GENERATOR" CONTENT="Mozilla/4.04 [en] (X11; U; SunOS 4.1.3_U1 sun4m) [Netscape]">
   <META NAME="sandia.approved" CONTENT="SAND99-1376">
   <META NAME="author" CONTENT="karen devine, kddevin@sandia.gov">
   <TITLE>Zoltan Developer's Guide:  Quality Program</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF">

<div ALIGN=right><b><i><a href="dev.html">Zoltan Developer's Guide</a> |
<a href="dev_dist.html">Next</a> | <a href="dev_intro_coding.html">Previous</a></i></b></div>


<H2>
<A NAME="SQE"></A>Zoltan Quality Assurance</H2>
This document describes the Software Quality Assurance (SQA)
policies and procedures used in the Zoltan project.  Zoltan developers
at Sandia or under contract to Sandia are required to follow these
software development policies.



<blockquote>
<a href="#Quality_Policy">Quality Policy</a>
<br><a href="#Quality_Definition">Quality Definition</a>
<br><A href="#Defect_Classification">Classification of Defects</a>
<br><A href="#Release_Policy">Release Policy</a>
<br><A href="#Quality_Tools">Software Quality Tools</a>
<br><A href="#Quality_Processes">Software Quality Processes</a>
<br><A href="#Zoltan_ASC">Zoltan&acute;s implementation of the ASC Software Quality Engineering Practices</a>
</blockquote>



<H3>
<A NAME="Quality_Policy"></A>Quality Policy
</H3>
<p>
Sandia&acute;s ASC Quality Management Council (AQMC) developed and manages the
Quality Assurance Program (QAP) for Sandia&acute;s ASC program.  The AQMC chartered
the development of the <I>Sandia National Laboratories Advanced Simulation and Computing
(ASC) Software Quality Plan, Part 1: ASC Software Quality Engineering Practices, Version 2.0</I>
document (SAND 2006-5998) as the practical SQA guidance for projects like Zoltan.
A companion document, <I>Sandia National Laboratories Advanced Simulation and Computing
(ASC) Software Quality Plan, Part 2: Mappings for the ASC Software Quality Practices</I> (SAND 2006-5997),
shows how these practices satisfy corporate policies including CPR001.3.6, Corporate
Software Engineering Excellence, and DOE/NNSA orders 414.1C and QC-1 rev 10.
<p>
The Zoltan project is committed to a program of quality improvement in compliance
with the <I>ASC Software Quality Engineering Practices</I> document.  The Zoltan Team Leader is the owner
of the Zoltan quality system. Zoltan developers at Sandia or under contract to Sandia
are required to follow these software development practices. The Zoltan team shall
participate in all reporting processes, audits, and assessments as directed by the AQMC.  

<H3><A NAME="Quality_Definition"></A>Quality Definition</H3>
<p>
QC-1 rev 10 defines quality as &quot;the degree to which customer requirements are met.&quot;
<p>
The Zoltan project accepts the following definition of quality:
&quot;the totality of characteristics of a product or service that bear
on its ability to satisfy stated or implied needs.&quot; This is Juran&acute;s 
&quot;fitness for use&quot; definition of quality (ANSI/ASQC A8402-1994.)
This superior definition of quality fully satisfies the QC-1 rev 10 definition.
This definition is also more useful in a research environment where the requirements are
derived from a research proposal rather than directly from customers and end users.

<H3><A NAME="Defect_Classification"></A>Classification of Defects</H3>
The Zoltan project accepts the following system of classification of
defects:
<BLOCKQUOTE>
<b>Critical:</b>  A defect that could lead to loss of life,
significant environmental damage, or substantial financial loss.
<br><b>Major:</b>  A non critical defect that significantly
impacts  Zoltan's fitness for use.
<br><b>Minor:</b>  A (non critical, non major) defect that
reasonably impacts Zoltan's fitness for use.
<br><b>Incidental:</b>  Any other defect which does not
reasonably reduce Zoltan's fitness for use.
</BLOCKQUOTE>
<p>

<H3><A NAME="Release_Policy"></A>Release Policy</H3>
<p>
Only the Zoltan team leader may authorize (certify) a release. 
The Zoltan team leader shall not release software with 
any known critical or major defects. 
User registration shall allow the Zoltan team to 
notify all Sandia and ASC users and to recall their
defective software if a critical or major defect 
is discovered after release. 
The Zoltan team leader may determine that it is
acceptable to release software with known minor or incidental defects.
<p>

<H3><A NAME="Quality_Tools"></A>Software Quality Tools</H3>
<p> 
Because of the small scale of the Zoltan Project, only a few, simple tools
are required for use by Zoltan developers:
<BLOCKQUOTE> 
<b>CVS:</b>  maintains code, documentation, meeting
notes, emails, and QA program artifacts;
<br><b>Purify, PureCoverage, Quantify (Rational), Valgrind, gdb:</b> 
for dynamic code testing, coverage measurements, and performance analysis;
<br><b>Bugzilla:</b>  tracks bugs, requests for changes,
and enhancements;
<br><b>Mailman:</b>  creates email lists to automatically
notify users by area(s) of interest;
<br><b>Makefiles:</b>  ensures proper compilation and linking
for all supported platforms; and
<br><b>Zoltan Test Script:</b> runs 
integration, regression, release and acceptance testing.
</BLOCKQUOTE>

<H3><A NAME="Quality_Processes"></A>Software Quality Processes</H3>
<p> 
<b>Bug Reporting, Issue Tracking, Enhancement Requests:</b>  All of these
items are now directly entered into Bugzilla by developers and users.
This &quot;process&quot; is built into the tool. Detailed instructions
for using Bugzilla are found on the Zoltan web page. Bugzilla also
provides query and report features for tracking the status of entered items;.
<p>
A process is defined as a sequence of steps performed for a given
purpose (IEEE Std. 610.12.)  Zoltan&acute;s other processes are defined as
checklists because checklists are one of the seven fundamental quality tools. 
These checklists are also the primary artifact created when following a process.
Currently the following processes are defined: 

<BLOCKQUOTE>
<b>Development:</b> (not currently used) defines the software development 
process including
requirements, design, implementation, testing, reviews, and approvals;
<br><b>Release:</b> defines the release process including testing requirements
and creation of the release product;
<br><b>Request:</b> defines the process of
capturing user requests for new features;
<br>&nbsp;&nbsp;Note: this process is now obsolete. Request processes in progress
may continue until complete but new requests should use Bugzilla;
<br><b>Requirement:</b>  the process of capturing user comments that
may become requirements after review and approval;
<br>&nbsp;&nbsp;Note: this process is now obsolete. Requirement processes in progress
may continue until complete but new requirements should use Bugzilla;
<br><b>Review:</b>  defines the materials reviewed prior to acceptance 
for Zoltan release;
<br>&nbsp;&nbsp; Note: Developers are encouraged to use Bugzilla to enter the
specific review process rather than use the Review checklist. At this time this
is an trial effort and either method may be used.
<br><b>Third Party Software:</b>defines the steps required to obtain, manage,
use, and test for software created outside of Zoltan and the ASC program; and
<br><b>Training:</b> defines the material a new developer must read, required
skills to demonstrate and computer accounts that must be obtained.
</BLOCKQUOTE>

<p>
Zoltan's software quality process checklists define how work may be performed,
including process ownership, authorization to perform, activities and their
sequence (when sequencing is required), process instructions, metrics, and
identification of who performed each activity. 
<p>
The only allowed source for process checklists is Zoltan's CVS repository
in the SQA_templates directory (under Zoltan_Internal.)  A Zoltan developer
initiates a process by obtaining the current CVS version of the process, renaming
it, and committing the renamed process checklist back into CVS in an appropriate
directory on the same day. The process may continue under this committed version
even if its original process is later superseded unless specifically requested by
the Team Leader. After one or more activities are completed, the process
checklist is updated to reflect the results and committed back to CVS (with
appropriate comments.) A process is completed when all required activities
are completed including reviews and approvals (as necessary), and committed to CVS.
The final CVS comment should indicate that the process is complete.
<p>
<H3><A NAME="Zoltan_ASC"></A>
Zoltan&acute;s implementation of the <I>ASC Software Quality Engineering Practices</I></H3>
<p> 
The following is brief description <b> for Zoltan developers</b> about the Zoltan
project&acute;s implementation of the <I>ASC Software Quality Engineering Practices</I> (SAND 2006-5998):
<p>
<b>PR1. Document and maintain a strategic plan.</b><br>
The Zoltan web page has a direct hyperlink to the Zoltan Project Description
defining its mission and philosophy. The Zoltan project has a strong association
with the Trilinos project to share in the development of common software
engineering practices and sharing of appropriate tools and experience.
<p>
<b>PR2. Perform a risk-based assessment, determine level of formality
and applicable practices, and obtain approvals.</b><br>
The Zoltan project has an approved level of formality (medium) for its
deliverable software.  Its biggest technical risk results from providing
parallel solutions to NP hard partitioning problems.  Technical risks are
mitigated by collaborations within Sandia and internationally.  The most
significant non-technical risk is the conflicting priorities of Zoltan
developers working on many other projects simultaneously.
<p>
<b>PR3. Document lifecycle processes and their interdependencies, and
obtain approvals.</b><br>
The Zoltan project follows the <I>Trilinos Software Lifecycle Model</I>
(SAND 2006-6929).  It also follows the ANSI/ASQ Z1.13-1999 standard
<I>Quality Guidelines for Research</I> which is compatible with the research
phase in the Trilinos Lifecycle model.
<p>
<b>PR4. Define, collect, and monitor appropriate process metrics. </b><br>
The Zoltan project is committed to comply fully with the new and evolving
AQMC requirements for collecting and reporting &quot;defect&quot; metrics.
Other metrics determined by Zoltan&acute;s continual process improvement process
(<b>PR 5</B>) will be implemented.
<p>
<b>PR5. Periodically evaluate quality problems and implement process
improvements.</b><br>
The Zoltan project has built the Deming/Shewhart process improvement
cycle PDCA (Plan, Do, Check, Act) into all of its process checklists.  This is
the most effective process improvement technique known.  It is recommended
by ISO 9001:2000.
<p>
<b>PR6. Identify stakeholders and other requirements sources.</b><br>
The Zoltan project&acute;s primary stakeholders are the ASC applications using
Zoltan including SIERRA, ACME, ALEGRA/NEVADA, XYCE, and Trilinos.
<p>
<b>PR7. Gather and manage stakeholders&acute; expectations and requirements.</b><br>
The Zoltan project&acute;s primary input from ASC applications&acute; expectations and
requirements are via their communication of Zoltan&acute;s role in meeting their
ASC milestones.  Since Zoltan is an &quot;enabling technology,&quot; these requirements
are broadly stated performance improvement needs.  The Zoltan team actively anticipates
and develops load balancing software for the future needs of the Sandia research community
before they actually become formal requirements.
<p>
<b>PR8. Derive, negotiate, manage, and trace requirements.</b><br>
Zoltan project requirements normally derive from its funded research proposals
which state research goals. This is a normal procedure in a research
environment (see ANSI/ASQ Z1.13-1999).  Periodic and final reports document
the success in meeting these research goals.
<p>
<b>PR9. Identify and analyze risk events.</b><br>
All Zoltan developers should report any new or changed risks via the zoltan-dev
email target for evaluation by the Team Lead.
<p>
<b>PR10. Define, monitor, and implement the risk response.</b><br>
The Zoltan team will create a corrective action plan whenever any condition
threatens to adversely impact the Zoltan project resources or schedule.
<p>
<b>PR11. Create and manage the project plan.</b><br>
ANSI/ASQ Z1.13-1999 states that the research proposal is equivalent to a
project plan in a research environment. The Team Leader assigns responsibilities,
deliverables, resources, and schedules in order to manage the project.
<p>
<b>PR12. Track project performance versus project plan and implement
needed (corrective) actions.</b><br>
The Team Leader periodically tracks responsibilities, deliverables, resources,
and schedules in order to manage the project.
<p>
<b>PR13. Communicate and review design.</b><br>
The Zoltan architecture is fully documented in the Zoltan Developer&acute;s Guide.
New features are originally documented and reviewed in team discussions to
the zoltan-dev email target. Prior to release, the design documentation is
finalized in both the Zoltan Developer&acute;s Guide and the Zoltan User&acute;s Guide.
<p>
<b>PR14. Create required software and product documentation.</b><br>
Developers will follow the Zoltan Development Process Checklist.
<p>
<b>PR15.  Identify and track third party software products and follow
applicable agreements.</b><br>
Developers will follow the Zoltan Third Party Software Process Checklist.
<p>
<b>PR16. Identify, accept ownership, and manage the assimilation of other
software products.</b><br>
Not applicable since Zoltan does not &quot;assimilate&quot; third party software.
<p>
<b>PR17. Perform version control of identified software product artifacts.</b><br>
All software and process artifact are under maintained CVS as early as reasonable
after their creation.
<p>
<b>PR18. Record and track issues associated with the software product.</b><br>
Developers will use Bugzilla to record and track issues.
<p>
<b>PR19. Ensure backup and disaster recovery of software product artifacts.</b><br>
Nightly backups, periodic offsite backups, and disaster recovery are services
provided by the CSRI computer support staff. Disaster recovery has been successfully
performed from real problems.
<p>
<b>PR20. Plan and generate the release package.</b><br>
Developers will follow the Zoltan Release Process Checklist.
<p>
<b>PR21. Certify that the software product (code and its related artifacts) is
ready for release and distribution.</b><br>
The Zoltan Team Leader will certify any version of Zoltan for release via an
email to zoltan-dev target.
<p>
<b>PR22. Distribute release to customers.</b><br>
Zoltan files are released via a download from the Zoltan web site. The Zoltan
Team Leader will make the download available after certification. (Research
versions of the Zoltan software are directly available to collaborators for
development.)
<p>
<b>PR23. Define and implement a customer support plan.</b><br>
(See <b>PR 6</b> for a list of ASC stakeholders.) The Zoltan team provides one-on-one
training whenever requested and quickly responds to any user complaint.
<p>
<b>PR24. Implement the training identified in the customer support plan.</b><br>
See <b>PR 23</b> above.  If additional training is ever requested, the Zoltan project
will piggy back on the annual Trilinos Users Group meeting with a training
session on using Zoltan.
<p>
<b>PR25. Evaluate customer feedback to determine customer satisfaction.</b><br>
<p>
<p>
<b>PR 26 Develop and maintain a software verification plan.</b><br>
Developers are expected to create new tests for the Zoltan test suite when
new features are added to Zoltan.
<p>
Currently, a new test framework based on FAST/EXACT is being implemented.
Documentation about this test framework is under preparation.  A process
checklist will be developed around the steps required to add new tests to
the suite and to run the suite.
<p>
<b>PR27. Conduct tests to demonstrate that acceptance criteria are met and to
ensure that previously tested capabilities continue to perform as expected.</b><br>
This practice is a subset of the Zoltan Release Process Checklist.
<p>
<b>PR28. Conduct independent technical reviews to evaluate adequacy with respect
to requirements.</b><br>
Developers will follow the Zoltan Review Process Checklist. ANSI/ASQ Z1.13-1999
states that the peer reviewed publications and conference presentations are a normal
form of technical review in the research environment.
<p>
<b>PR29. Determine project team training needed to fulfill assigned roles 
and responsibilities.</b><br>
New developers will follow the Zoltan Training Process for new team members.
<p>
<b>PR30. Track training undertaken by project teams.</b><br>
Zoltan developers are encouraged to participate in the annual Trilios Users Group
(TUG) meeting which provides sessions for SQA/SQE training to developers.
Attendance records are kept for this event and for any Zoltan team meetings that
provide training. Sandia provides many other opportunities for training including
formal courses and periodic internal software developers conferences.  External
conferences (e.g., IPDPS and SIAM) are counted as technical training.

<p>
<HR WIDTH="100%">
<BR>[<A HREF="dev.html">Table of Contents</A> | <A HREF="dev_dist.html">Next:
Zoltan Distribution</A> | <A HREF="dev_intro_coding.html">Previous:
Coding Principles in Zoltan</A>
| <a href="https://www.sandia.gov/general/privacy-security/index.html">Privacy and Security</a>]
</body>

</html>
